TRW  Systoms  Engineering  & 
Development  Division 

TRWTS-89  01 


ADrA242  965 


TRW.TS.89.D1  %  ^  1991  y 

Q  IMF 

Software  Project  Management  Using  Effective 
Process  Metrics:  The  CCPDS-R  Experience 


Don  Andres 

November  1989 


ipp 

pkh 

PiP 

III 

ppR 

Wm^ 

HI  ri 

....  /ri-MKNT  A 

Apptovod  lor  public  rol«a»«: 
DiaUibution  UnHmltod 


37-13705 

hill  iea<> 


ogy 

ogy 


¥iiiiirr 


logy  5ien 
ogy  Seri 
logy  Seri 

logy 


logy 
logy 
oiogy  Series 


SOFTWARE  PROJECT  MANAGEMENT 
USING  EFFECTIVE  PROCESS  METRICS: 
THE  CCPDS-R  EXPERIENCE 


Donald  Andres 
TRW  Defense  System  Group 
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ABSTRACT 


The  paper  captures  the  project  management  experience  of  using  process  metrics  to  mea¬ 
sure  the  development  of  a  large  software  system  for  the  U.S.  government.  The  Command 
Center  Processing  and  Display  System-Replacement  (CCPDS-R)  program  is  a  development 
effort  consisting  of  three  subsystems  and  over  600,000  lines  of  Ada  source  code  executing  in 
a  distributed  Digital  Equipment  Corporation  VAX  VMS  environment.  The  initial  subsystem 
called  the  Common  Subsystem,  consists  of  over  300,000  lines  of  Ada  source  code  and  is  2  years 
into  development  with  a  successful  Critical  Desi^  Review  (CDR)  held  at  month  25  of  the  con¬ 
tract.  Utilizing  the  new  Ada  process  model,  over  280,000  lines  of  Ada  source  code  have  been 
developed  and  integrated  into  a  working  demonstration  on  the  operational  hardware.  This 
approach  has  permitted  the  validation  of  the  software  design,  allowed  early  analysis  and  reso¬ 
lution  of  the  software  performance  issues,  and  provided  extensive  insight  into  the  usability  and 
flexibility  of  the  system  for  the  user  community.  ^ 


Tliis  first  program  use  of  the  incremental,  demonstration  based  process  model  with  re¬ 
lated  software  metrics  has  been  totally  successful.  The  technology  now  is  being  transferred  to 
othw-projjctswithin  the  company.  The  U.S.  Government  acquisition  agency  has  designated 
CCPDS-R  asitT^flagship  program’Vnd  is  basing  new  acquisitions  on  the  approach  utilized  by 
the  CCPDS-R  program. 


PROJECT  BACKGROUND 

The  Command  Center  Processing  and  Display  System  Replacement  (CCPDS-R)  program 
will  provide  display  information  used  during  emergency  conferences  by  the  National  Command 
Authorities;  Chairman,  Joint  Chiefs  of  Staff;  Commander  in  Chief  North  American  Aerospace 
Command;  Commander  in  Chief  United  States  Space  Command;  Conunander  in  Chief  Strate¬ 
gic  Air  Command;  and  other  nuclear  capable  Commanders  in  Chief.  It  is  the  missile  warning 
element  of  the  new  Integrated  Tactical  Warning  and  Assessment  system  wchitecture  developed 
by  North  American  Aerospace  Defense  Command/ Air  Force  Space  Command. 

The  CCPDS-R  program  is  being  procured  by  Air  Force  System  Command  Headquarters 
Electronic  Systems  Division  (ESD)  at  Hanscom  AFB  and  was  awarded  to  TRW  Defense  Systems 
Group  in  Jxme  1987.  TRW  will  build  three  subsystems.  The  first,  identified  as  the  Common 
Subsystem,  is  25  months  into  development.  The  Common  Subsystem  consists  of  over  300,000 
source  lines  of  Ada  with  a  softare  development  schedule  of  38  months.  It  will  be  a  highly 
reliable,  near  resd-time  distributed  system  with  a  sophisticated  User  Interface  and  stringent 
performance  requirements.  It  will  be  implemented  entirely  in  Ada. 
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The  approach  to  software  development  on  CCPDS-R  is  unique  to  the  industry.  Extensive 
tailoring  of  DoD-STD-2167,  by  TRW  and  the  government  as  a  team,  allowed  many  innovative 
techniques  and  the  orchestration  of  a  successful  Ada  software  development  program.  These 
innovations  already  have  been  proposed  on  other  TRW  developments  and  cotild  well  become  a 
DoD  and  industry  standard  for  Ada  software  development. 

The  specific  features  of  the  new  Process  Model  are  described  in  another  paper,  “TRW’s 
Ada  Process  Model  for  Incremental  Development  of  Large  Software  Systems”  [Royce  1990].  In 
summary,  the  Ada  process  Model  is  a  uniform  application  of  incremental  development  coupled 
with  a  demonstration-based  approach  to  design  review  for  continuous  and  insightful  thread 
testing  and  risk  management.  The  use  of  Ada  as  the  life  cycle  language  for  design  evolution 
provides  a  vehicle  for  uniformity  and  a  basis  for  consistent  software  progress  metrics.  Ada 
has  proven  also  to  be  a  significant  benefit  in  the  incremental  development/integration/test  ap¬ 
proach,  particiilarly  in  enabling  rapid  integration  of  software  from  multiple  CSCIs. 

The  CCPDS-R  Software  test  approach  is  described  in  a  paper,  “Incremental  Software  Test 
Approach  for  DoD-STD-2167A  Ada  Projects”  (Springman  1989].  The  test  approach  has  been 
modified  and  enhanced  significantly  as  both  TRW  and  the  customer  better  understand  the  test 
requirements  and  implications  of  specific  test  techniques.  Tailoring  of  the  approach  is  necessary 
as  experience  is  gained  from  earlier  test  phases. 

This  paper  describes  the  management  implications  of  operating  under  the  new  software  de¬ 
velopment  process.  The  primary  emphasis  and  features  provide  early  visibility  into  the  process 
with  extensive  prototyping,  incremental  builds  and  early  demonstrations.  The  process  is  then 
moiutored  and  managed  by  use  of  software  process  metrics  tailored  to  the  CCPDS-R  program. 

o  Management  Characteristics 

CCPDS-R  employ  a  number  of  innovative  management  and  technical  methods  that  are 
intended  to  maximize  software  development  productivity  while  minimizing  development  and 
test  risks.  Many  of  these  methods  take  advantage  of  Ada  software  engineering  techniques,  and 
depart  from  the  traditional  software  development  “waterfall”  process  in  favor  of  an  approach 
closer  to  the  “spiral”  model  [Boehm  85].  The  CCPDS-R  approach  emphasizes  continuous  early 
design  integration  and  demonstration  (Figure  1).  TRW  has  incorporated  the  Ada  Process 
Model  in  the  CCPDS-R  software  planning  documents,  including  the  Softwsu’e  Development 
Plan  (SDP)  and  the  Software  Standards  and  Procedures  Manual  (SSPM).  The  model  is  based 
on  incremental  development  and  extensive  prototyping,  and  is  being  refined  as  experience  is 
gained  during  the  development  of  the  CCPDS-R  software.  The  particular  innovations  of  the 
CCPDS-R  approach  are  discussed  briefly  herein.  These  innovations  have  helped  formulate  a 
good  set  of  process  metrics  to  measure  software  status. 

o  Two  Pass  Design  Approach 

The  CCPDS-R  approach  is  a  two  pass  design  philosophy  (Figure  2);  the  first  pass  is  proto¬ 
typing  to  demonstrate  feasibility  and  the  second  pass  formally  captures  the  design  recommended 
by  the  prototyping  results.  To  employ  this  approach  most  productively,  easily  reconfigurable 
software  architecture  components  are  needed  to  rapidly  integrate  applications  components  to 
determine  software  fimctionsdity  and  projected  performance.  On  CCPDS-R,  these  components 
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Figure  1:  Common  Subsystem  Software  Development  and  Integration 

are  provided  by  the  Network  Architecture  Services  (NAS)  CSCI  and  the  Software  Architecture 
Skeleton  (SAS)  which  encapsulates  the  top  level  executable  components  and  their  interfaces 
[Royce  89]. 

The  management  ramiftcations  of  this  approach  are  associated  with  time  and  technical 
risk.  It  generally  takes  longer  to  complete  the  design  and  code  phase,  generate  the  related 
documentation,  and  prepare  the  software  for  “turnover”  to  the  I&T  group.  This  time  is  offset, 
however,  by  lowering  the  technical  risk  of  imdetected  design  problems  and  performance  issues. 
The  sooner  problems  are  discovered,  the  cheaper  they  are  to  fix.  A  design  flaw  uncovered  as  a 
result  of  prototyping  should  not  be  regarded  as  a  “failure”,  but  rather  as  a  positive  result  that 
will  steer  the  design  in  the  proper  direction  much  earlier.  The  management  focus  of  this  new 
process  is  to  identify  and  resolve  all  major  software  risk  issues  by  PDR. 

The  incremental  development  approach  has  numerous  management  benefits;  it  breaks  up 
a  large  software  development  job  into  manageable  entities.  This  minimizes  the  communication 
complexity  between  team  members  since  fewer  individuals  are  working  in  the  development  of 
any  one  buld.  Management  must  focus  on  contents  of  the  builds  initially  to  ensure  cohesion  in 
contents  and  then  that  functional  capability  (and  correspondingly  the  development  effort)  does 
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Figure  2:  Software  Development  Approach- Common  Subsystem 


not  slip  to  later  builds.  An  indirect  effect  of  this  process  is  a  highly  motivated  software  engi¬ 
neering  and  development  organization.  They  strive  to  produce  an  excellent  product  because  of 
the  near  term  visibility  of  their  product  -  software  code. 

o  Early  Demonstrations 

In  concert  with  the  early  prototyping  philosophy,  tangible  design  progress  is  shown  to 
CCPDS-R  program  management  and  the  government  early  and  often  during  the  design  process 
(Figure  2).  On  past  programs,  where  design  progress  was  measured  by  how  much  documen¬ 
tation  and  paper  analysis  was  produced  in  support  of  PDR  and  CDR,  the  paper  reviews  were 
usually  judged  successful,  with  little  visibility  into  design  feasibility  or  potential  problems.  To 
correct  this,  formal  demonstrations  are  scheduled  for  the  Customer  and  Users  at  key  points  in 
the  design  process.  Items  to  be  demonstrated  include  user/system  interfaces,  functional  soft¬ 
ware  capabilities,  and  protot3rped  solutions  for  high  risk  items.  The  demonstrations  are  nattiral 
extensions  of  design  activities,  mininuzing  the  generation  of  formal  documentation  and  demo- 
specific  software.  These  demonstrations  provide  early  visibility  in  selected  design  areas  so  as 
to  obtain  early  feedback,  not  to  verify  requirements  or  performance.  Management  attention  is 
required  to  ensure  that  the  government  and  the  contractor  stick  to  the  purpose  of  the  demon¬ 
strations.  This  approach  has  resulted  in  125,000  source  lines  of  Ada  code  being  developed, 
integrated,  and  demonstrated  at  PDR  (month  16  of  current  contract)  and  280,000  Ada  source 
lines  at  CDR  (month  25).  These  demonstrations  included  critical  components  executing  under 
peak  load  to  provide  tangible  evidence  that  TRW’s  design  would  meet  the  stringent  CCPDS-R 
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Figure  3:  Reusable  Software 


performance  requirements.  Informal  demonstrations  sure  given  as  part  of  design  walkthroughs 
or  status  reviews.  The  demonstration  audiences  include  senior  engineering  personnel  from  other 
programs  in  TRW  to  promote  technical  feedback  and  reuse  of  by  other  programs. 

o  Reusable  Soflwcu'e  Components 

Ada  encourages  modularity  in  the  software,  which  promotes  reuse  of  components.  For 
example,  on  CCPDS-R,  the  entire  NAS  CSCI  is  essentially  an  application-independent  set  of 
components  reusable  by  any  DEC  VAX/VMS  application  [Royce  1989].  On  CCPDS-R,  the 
software  developed  imder  the  basic  contract  for  the  Common  Subsystem  will  be  heavily  reused 
for  subsequent  Processing  Display  Subsystem  (PDS)  and  SAC  Subsystem  options,  because  the 
basic  software  is  being  designed  for  reuse.  (Figure  3) 

Particular  management  attention  is  necessary  to  ensure  the  maximum  reusability  of  the 
software.  While  this  is  aided  by  standards  and  features  of  Ada,  the  concept  of  reusability  must 
continually  be  a  topic  for  discussion  at  technical  interchanges  and  design  reviews.  The  govern¬ 
ment  and  development  team  must  be  kept  aware  of  the  reusability  benefits  so  as  to  tailor  the 
various  subsystems  requirements  to  enhance  the  potential  for  reusable  software. 


o  Use  of  Schedule  Tools 
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CCPDS-R  has  numerous  milestones  and  hierarchies  of  schedules,  which  are  used  by  man¬ 
agement  and  developers  at  aU  levels  of  the  program.  It  is  essential  to  employ  a  scheduling 
methodology  that  is  tool  based  in  order  to  keep  the  process  of  regularly  statusing  and  updating 
the  schedules  manageable.  CCPDS-R  is  using  the  VISION  scheduling  system  with  a  compan¬ 
ion  graphics  generator.  The  scheduling  organization  has  integrated  all  of  the  C/SCSC  data, 
WBS  elements,  and  detailed  activity  networks  and  milestones  into  a  tiered  set  of  schedules  that 
satisAes  Government  and  TRW  needs.  All  schedules  are  statused  monthly  with  the  earned  value 
cost  data  driving  the  schedule  update  process  relatively  automatically.  This  provides  an  easy 
mechanism  to  compare  cost /schedule  data  with  other  process  metrics.  In  addition,  detailed 
weekly  schedules  are  generated  showing  activities,  milestones  and  products  for  each  performer. 
These  are  statused  at  least  monthly,  and  serve  to  create  schedule  awareness  throughout  the 
program  to  the  worker  level. 

o  Risk  Management 

Software  risk  management  is  a  continuous  activity  on  any  large  program.  CCPDS-R’s 
incremental  development  approach  combined  with  extensive  early  prototyping  is  a  continuous 
risk  management  strategy.  In  general,  every  program  should  generate  a  Risk  Management  Plan 
that  identiRes  risks  and  concentrates  on  the  top  problem  sureas  and  mitigation  procedures  to  be 
followed.  CCPDS-R  has  a  Risk  Review  Board,  comprised  of  representatives  from  all  program 
areas,  that  meets  at  least  monthly  to  identify,  assess  sind  resolve  risk  items.  All  risk  items  are 
identlRed  as  early  as  possible,  documented,  and  assigned  to  a  speciRc  individual  to  work,  along 
with  a  speciRc  plan  for  mitigating  the  risk.  We  regularly  review  the  risk  items  to  assure  that 
all  possible  avenues  for  solutions  are  explored. 


SOFTWARE  DEVELOPMENT  MANAGEMENT  INDICATORS  (METRICS) 

A  set  of  management  metrics  has  been  developed  by  TRW  and  the  government  to  facilitate 
greater  insight  into  the  software  development  process.  This  is  a  direct  fallout  of  a  government 
directive  by  General  Skantz  requiring  use  of  software  process  metrics.  The  CCPDS-R  metrics 
were  planned  and  implemented  to  minimize  productivity  impacts.  Metrics  collection  standards 
were  incorporated  into  the  Software  Standards  and  Procedures  Manual  early  in  the  program 
(Month  3)  to  impose  metrics  collection  through  software  standards.  Automated  tools  were  de¬ 
veloped  for  collecting  many  of  the  detailed  metrics  directly  from  the  evolving  Ada  design  Rles. 

One  of  the  key  features  of  the  homogeneous  Ada  life  cycle  representation  (i.e.,  Ada  as  a 
design  language)  [Royce  1990]  is  the  consistency  of  metrics.  Throughout  the  life  cycle,  the  same 
metrics  can  be  evaluated  to  ascertain  progress,  product  change  and  product  quality.  Since  Ada 
program  units  are  being  developed  in  the  design  phase,  their  metrics  can  be  compared  against 
planning  estimates  to  evaluate  progress  and  trends.  This  gives  numagement  objective  insight 
early  in  the  life  cycle  and  allows  reaction  to  any  changes  which  are  considered  off-nominal  before 
they  result  in  undesirable  side  effects. 

By  providing  continuous  Software  Problem  Report  (SPR)  metrics  against  all  conRgura- 
tion  controlled  components,  component  reUabillty  and  quality  can  be  assessed.  This  provides 
early  indicators  of  which  components  require  further  work.  These  metrics  are  tracked  during 
the  management  of  the  project,  with  planned  value  profiles  measured  versus  thresholds  which 
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alert  management  that  redirection/further  evaluation  is  necessary.  Under  configuration  man¬ 
agement,  design  problems  and  fixes  are  formally  tracked.  The  record  of  problems  and  solutions 
provides  substantial  data  for  predicting  future  performance  and  product  reliability. 

The  following  is  a  list  of  metrics  which  are  gathered  and  reported  monthly  on  the  CCPDS-R 
project. 

•  CSCI  and  Subsystem  Characteristics 

1.  Number  of  Ada  source  lines  (Figure  4)  by  CSCI  and  total  subsystem 

2.  Number  of  Software  Development  Files  (SDFs)  (Figure  6) 

3.  Complexity 

•  Development  Progress  by  Build,  by  CSCI  and  for  the  whole  subsystem  (Figure  6  and 
Figure  7) 

1.  Design  Progress;  ratio 

2.  KSLOC  Standalone  tested 

3.  KSLOC  Integrated 

•  Effort  Expenditure 

1.  Cost  variances.  Schedule  variances  (By  CSCI  and  Build) 

2.  Effort  expended  by  Development  activity  (Preliminary  Design,  Testing,  planning, 
etc.) 

3.  Staffing  (Actual  vs  Plaimed)  (Figure  8) 

•  Software  Architectiire  Stability 

1.  Number  of  Nodes,  Ada  Main  Programs,  Ada  Tasks,  Sockets  (Top  Level  Message 
Interfaces) 

•  Program  Volatility  (ECP  Impacts) 

•  Action  Item  Closure  History 

•  SPR  History:  Number  of  SPRs  by  Build,  by  CSCI  and  for  the  whole  system  (Figure  11) 

•  Software  Testing 

1.  Subsystem  test  progress  (Figure  9) 

2.  Number  of  test  procedures  (estimated  and  completed) 

3.  Niunber  of  SRS  requirements  (existing  and  verified)  (Figure  10) 

•  Reliability  (Software  failures  following  a  baseline  turnover)  (Figure  12) 

•  Resource  Utilization 

1.  Host  development  environment 

2.  Target  hardware  utilization 
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o  Management  Use  of  Metrics 


Our  experience  with  metrics  collection  and  reporting  has  been  very  positive.  Trends  and 
issues  have  been  explicitly  observable,  stimulating  detailed  questions  and  early  resolution  of 
potential  problems.  The  metrics  approach  has  also  promoted  more  uniform  communications, 
checks  and  balances  and  quantitative  explanations  for: 

1.  Customer/ Contractor  interaction 

2.  Customer/Higher  Government  Authority  Discussions 

3.  Project /Corporate  Management  Review 

4.  Internal  Software  reviews 

5.  Software/Project  Manager  Review 

Highlights  of  these  metrics  will  be  discussed  herein  concentrating  on  the  current  progress 
(Month  24)  of  the  program  to  illustrate  the  effective  use  of  the  metrics  as  management  mea¬ 
surement  tools. 

The  software  size  history  (Figure  4)  depicts  the  software  size  in  terms  of  Ada  source  lines 
of  code  at  contract  award  versus  the  current  size  estimate.  The  graphic  depicts  the  estimate 
of  the  software  size  as  it  has  changed  based  on  monthly  estimates.  Normally,  a  graph  as  illus¬ 
trated  in  Figure  4  would  indicate  disaster  since  the  size  has  doubled  in  two  years.  However,  on 
CCPDS-R,  we  adopted  the  Ada  COCOMO  method  of  counting  source  lines  of  code  midway 
through  the  process  [Royce  1990]  and  [Boehm/Royce  1988).  This  resulted  in  the  largest  increase 
in  code  size  -  primarily  in  SSV.  Secondly,  we  opted  to  produce  some  of  the  mundane,  clerical 
software  with  tools  as  opposed  to  the  brute-force  coding  technique.  As  a  result,  a  software  tool 
set  sized  at  about  15,000  Ada  source  lines  of  code  produces  approximately  200,000  source  lines 
of  operational  code  (Table  l). 


Tool  Characteristics 

1  Operational  SW  Produced 

Name 

SLOC 

Dev.  MM 

CSCI 

Function 

SLOC 

SAS  Builder  Tool 

6500 

10 

SSV 

SW  Architecture  Skeleton 

Format  Build  Tool 

3500 

6 

DCO 

Display  Format  Tables 

ISkftililifl 

CCPDS-R  Message  Tool 

3500 

13 

CCO 

External  I/F  Validation 

Format  Outbound  Messages 

■  J  raTTTl  »7Tn7T!!l  »7T;  WTfflTW 

SGI  Message  Tjpe  Declarations 

■■ 

SSMS 

3,000 

Common  tc  SAC 

CCOMS 

3,000 

mM 

Forward  Users  Ic  Sensors 

6,000 

Message  10  Tool 

3100 

3 

\  SSV 

Data  Reduction/Msg  Print 

Total 

14,600 

30 

1  "Cookbook’*  Software 

202,000 

Table  1:  Tool  Produced  Software 
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The  overall  progress  for  each  subsystem  is  described  at  a  high  level  via  the  Development 
Progress  Simunary  (Figure  5).  This  single  metric  shows  the  development  schedule,  software 
size  and  progress  to  date  by  software  build.  The  progress  is  depicted  as  percent  complete  for 
design/ code  and  standalone  test  activities.  The  shading  indicating  overall  build  progress  repre¬ 
sents  a  combination  of  the  financial  cost/schedule  data  and  the  software  detailed  development 
metrics  data.  This  metric  gives  a  good  overall  picture  of  status  and  potential  problem  areas 
requiring  management  attention,  and  has  been  an  effective  means  of  conveying  status  to  higher 
levels  of  management.  The  next  level  of  detadl  for  software  development  progress  indicates 


Figtxre  5:  Development  Progress- Conunon  Subsystem 

the  status,  by  software  CSCI  for  each  build  (Figure  6).  The  total  number  of  software  devel¬ 
opment  files  is  tracked  as  well  as  the  total  estimated  source  lines  of  Ada  code  by  CSCI.  The 
designed/coded  status  is  indicated  by  percent  complete  (Ada/(Ada-|-ADL))  of  software  source 
code.  The  current  month  (CM)  column  indicates  the  progress  made  during  the  past  month  by 
CSCI.  This  is  helpful  to  determine  which  areas  are  falling  behind  or  making  little  progress.  The 
“tested”  column  shows  informal  stand  alone  testing  complete  as  a  percent  of  the  total  source 
lines  of  code.  The  “dociimented”  status  depicts  the  percent  of  software  development  files  with 
completed  documentation.  The  graphics  depict  the  actual  progress  versus  the  plan  for  the  total 
common  subsystem  software  for  designed,  coded,  standalone  tested,  and  dooimented.  This 
metric  allows  immediate  insight  into  progress  versus  the  plan.  This  same  metric  with  build 
specific  metrics  (Figure  7)  is  used  as  the  primary  measurement  by  management  to  track  the 
individual  builds  and  is  especially  useful  when  many  builds  are  in  development  simultaneously. 
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Figure  6:  Development  Progress- Common  Subsystem 
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Software  staffing  (Figure  8)  is  shown  as  personnel  versus  time  and  shows  actual  headcount 
as  well  as  planned  staffing  needs.  Additional  metrics  track  attrition  and  additions  on  a  monthly 
basis.  A  large  turnover  in  personnel  each  month  is  a  good  indicator  that  downstream  problems 
will  occur  due  to  the  added  training  and  “learning  curve”  required  to  assimilate  new  person¬ 
nel.  CCPDS-R’s  experience  to  date  has  been  very  positive  with  very  low  attrition  other  than 
planned  departures.  For  example,  the  large  change  in  Month  24  indicated  the  plaimed  addition 
of  test  personnel  and  design  engineers  leaving  the  project. 
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Figure  8:  Staffing  Profile  History  and  Addition/ Attrition  History 
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Figure  9;  Testing  Progress-Common  Subsystem 

The  overall  test  progress  for  each  subsystem  is  measured  in  a  single  graphic  metric  (Fig¬ 
ure  9)  similar  to  the  development  progress  metric.  This  graphic  is  divided  into  two  sections: 

•  Build  Integration  Testing  (BIT) 

•  Formal  Test  including  Stand  Alone  Testing  (SAT),  Engineering  String  Testing  (EST)  and 
Formal  Qualification  Testing  (FQT) 

The  BIT  progress  is  tracked  for  each  software  build  by  source  lines  of  code  and  progress  of 
test  cues  generated  and  executed.  The  measurement  is  percent  complete;  progress  assessment 
is  based  on  the  monthly  financial  data.  The  Formal  Test  progress  is  tracked  by  requirements 
verified  in  each  test  period  (SAT,  EST,  or  FQT)  and  progress  of  test  cases  generated  and  ex¬ 
ecuted.  There  is  an  overall  status  line  to  indicate  the  total  subsystem  progress  as  a  percent 
integrated  and  as  a  percent  verified. 

The  Software  Requirement  Verification  metric  (Figure  10)  provides  the  detailed  plan  and 
status  by  test  period  and  CSCI.  This  metric  contains  an  indicator  per  CSCI  per  test  period. 
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•  Common  Subsystem 
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Figure  10:  Verification  Testing  Progress 


M/N  where  M  is  the  number  of  requirements  aUocated  to  the  test  period  and  N  is  the  number 
of  requirements  successfully  verified  and  signed  off  by  the  government.  The  graphic  depicts  the 
progress  versus  the  plan  as  a  composite  of  the  total  requirements  to  be  verified  in  the  subsystem. 

One  of  the  benefits  of  the  incremental  development  process  is  the  ability  to  evaluate  soft¬ 
ware  quality  and  reliability  early  in  the  development  cycle.  A  measure  of  this  quality  can  be 
determined  by  the  number  of  Software  Problem  Reports  (SPRs)  per  CSCI  (Figure  11).  SPR 
statistics  are  gathered  by  CSCI,  priority,  and  category.  The  open  SPRs  are  addressed  at  the 
weekly  Software  Configuration  Control  Board  (SCCB)  to  review  status  and  approve  appropri¬ 
ate  action  for  resolution.  The  graphic  metric  tracks  SPRs  by  age  and  is  helpful  to  determine 
delinquent  SPRs  which  may  severely  impact  other  areas  of  the  program.  We  use  a  metric  of 
SPRs/month  scaled  to  1000  Source  lines  of  Ada  code  under  test  to  illustrate  software  quality 
and  reliability  (Figure  12).  The  CCPDS-R  average  rate  to  date  is  .3  SPRs  per  1000  SLOCs 
per  month,  which  is  well  below  TRW’s  experience  on  other  major  software  projects  at  the 
equivalent  stage  of  development. 
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Figure  11:  Software  Problem  Reports  (SPRs) 


Figure  12:  SPRs  per  KSLOC  per  Month 


LESSONS  LEARNED 


The  CCPDS-R  program  experience  using  the  new  Ada  Process  Model  and  software  metrics 
has  proven  to  be  a  successful.  The  project  has  reacted  well  to  the  new  approach  and  adapted  as 
necessary.  Likewise,  we  have  adapted  the  initial  process  model  to  provide  greater  efficiency  and 
improved  productivity.  The  following  aspects  of  managing  the  software  development  process 
are  working  well: 


•  The  early  preparation  and  planning  for  the  program.  This  was  accomplished  largely  dur¬ 
ing  the  concept  definition  phase  prior  to  the  fuU-sctile  development  and  focused  on  stan¬ 
dards,  tools,  schedules,  staffing  plans,  test  program  plans  and  early  development  of  the 
foimdation  components  -  NAS. 

•  The  general  approach  based  on  the  new  process  model.  This  includes  the  early  software 
builds  and  design  integration  resulting  in  software  being  integrated  largely  prior  to  turnover 
to  Integration  and  Test.  The  technical  quality  evaluation  that  is  being  performed  by  the 
software  engineering  and  software  test  organizations  augments  the  typical  software  quality 
assurance  activities. 

•  The  software  metrics.  The  software  metrics  have  proven  to  be  effective  management  mea¬ 
surement  tools  for  progress  and  quality  assessment. 

•  The  capability  demonstrations.  These  demonstrations  are  timely,  identify  technical  risk 
areas  early  and  focus  risk  attention,  force  integration  as  part  of  the  design  process  early, 
and  make  software  visible  to  customers,  users,  and  management. 

•  The  early  exercise  of  standards  fc  tools.  The  development  standards  and  tools  were  pro¬ 
duced  prior  to  the  start  of  the  full-scale  development  effort  allowing  the  software  team  to 
hit  the  ground  running  at  contract  start.  The  first  build  pioneered  the  SSPM  and  tool 
set  to  incorporate  lessons  learned  into  the  larger  later  builds. 

•  Adquate  schedule  for  PDR  and  CDR.  Scheduling  adequate  time  for  requirement  defini¬ 
tion  and  design  prototying  has  helped  to  produce  effective  demonstrations  at  these  major 
milestones  thereby  lowering  the  risk  to  the  I&T  schedule. 

•  Problem  identification/resolution.  The  software  team  made  numerous  mistakes  in  the  first 
10  months  of  the  project.  We  solved  these  problems  early,  flushed  out  the  inefficiencies 
in  the  new  process  and  substantially  reduced  rework  time. 

On  the  flip  side,  all  is  not  perfect.  There  are  some  areas  that  can  be  improved  as  the 
project  continues  into  the  two  new  subsystems. 

•  Computer  resources.  Ada  development  projects  require  extensive  computing  resources 
(CPUs,  memory,  disks).  This  is  especially  true  when  the  software  is  integrated  into  a 
large,  testable  entity.  Making  efficient  use  of  these  resources  is  a  difficult  task,  requiring 
detailed  planning  for  effective  use  of  computer  resources. 

•  Keep  design  out  of  the  requirement  specification.  We  were  somewhat  successful  in  this 
area;  the  project  still  has  too  much  design  detail  in  the  software  requirement  specifications. 
[Grauling  1989] 
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•  Everyone  must  follow  Process  Model:  Designing  and  coding  early  requires  total  commit¬ 
ment.  The  government  personnel  and  the  project’s  system  engineering  organization  must 
madce  timely  decisions,  especially  as  related  to  hardware  and  user  interfaces.  CCPDS-R 
uses  govemment/contractor  working  groups  to  make  these  decisions,  but  timeliness  could 
still  improve. 

•  Automated  Graphical  representation  of  design.  While  this  was  not  required  by  the  soft¬ 
ware  development  team,  it  is  a  definite  need  by  the  reviewers  of  Ada  design,  such  as, 
project  management,  system  engineering,  and  government  personnel.  The  development 
team  has  fotmd  that  Ada  itself  is  readable,  unambiguous,  and  sufficient  as  a  design/ development 
methodology. 

•  Software  Configuration  Control.  Our  configmation  control  concept  is  based  on  a  set  of 
FORTRAN  tools  adapted  to  work  in  an  Ada  development  environment.  These  are  not  well 
tuned  to  Ada  and  Ada  compiler  capabilities.  We  are  presently  investigating  improvements 
in  this  area. 

•  Test  program.  The  effectiveness  of  the  test  progress  is  yet  to  be  determined.  The  approach 
to  incremental  testing  seems  to  be  working.  The  government  imposed  military  standards 
and  data  item  descriptions,  which  impose  rigid  constraints,  can  lead  to  a  costly  test 
program.  [Springman  1989] 


SUMMARY 

The  success  of  the  CCPDS-R  Program  to  date  has  been  extraordinary.  The  software  de¬ 
velopment  team,  aU  levels  of  management,  and  the  customer/user  commimity  are  very  satisfied 
with  our  utilization  of  the  new  process  model  with  progress  metrics.  Managers  and  customers 
appreciate  the  added  insight  into  the  software  development  process  and  the  technical  risk  evalua¬ 
tion  coupled  with  the  approach.  The  software  development  and  test  team  enjoy  the  opportunity 
to  see  their  efforts  demonstrated  early  in  the  process.  This  indirectly  provides  strong  motivation 
to  do  well  and  has  allowed  them  to  have  more  time  to  focus  on  design  and  formal  test,  while 
minimizing  the  rudimentary  software  integration  process.  The  user  conununity  is  probably  the 
most  impressed  since  they  have  “seen”  their  ultimate  product  very  early  in  the  development 
cycle.  This  has  permitted  them  to  refine  their  “usability  ideas”  with  actual  hardware  and  soft¬ 
ware,  as  opposed  to  using  paper  concepts. 
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